For related art driving support apparatus which uses image pickup means attached to a vehicle, there is a method to change the viewpoint of a composite image. The method is disclosed for example in JP-A-58-110334. In the case of this related art apparatus, a composite image seen from a new viewpoint, for example from above, is created. In such a case, the distance from a vehicle and another object is proportional to the distance on the screen, so that it is easy to intuitively understand the actual distance. While such driving support apparatus is sometimes called a mobile unit image display system, the term driving support apparatus is used in the specification.
Operation of the related art example will be explained using FIGS. 31A to 31D. FIG. 31A is a schematic view to explain the relation between actual image pickup apparatus and a virtual viewpoint. In the figure, image pickup apparatus 1001 attached to a vehicle picks up the rear of the vehicle, mainly the road surface at a downward angle. Assuming that the direction of pixels 1002 where an image of image pickup apparatus is present picks up a point 1003 on a three-dimensional road surface, it is understood that the position seen from a virtual viewpoint above (virtual camera) 1004 corresponds to pixels in a direction 1005 on the screen of an image of the virtual camera. FIG. 31B shows the situation of a vehicle and a rear camera in an actual parking lot and a periphery of the parking lot. The image pickup apparatus 1001 picks up the image of the range of field 1006 in the parking lot. In this example, in particular the direction of pixels 1002 is directed to a point 1003 in the corner of white lines on the road surface. FIG. 31C is an image actually obtained from the image pickup apparatus 1001. The point 1003 in the corner of white lines on the road surface has its position converted and compounded to correspond to the pixels in the direction 1005 seen from the overhead virtual viewpoint 1004. FIG. 31D is a composite image. In the figure, the point 1003 is positioned on the road surface so that the relative position with respect to the vehicle is accurately reproduced. All the while lines in other parking sections are positioned on a road surface so that the relative position with respect to the vehicle is accurately reproduced on a composite image.
The driver of a vehicle can understand the relation between the vehicle and its surroundings by watching this composite image.
However, the related art technology has a problem. The problem will be described using FIGS. 32A to 32D.
FIG. 32A explains a case where an object positioned elsewhere than on a road surface, such as a bumper of a vehicle is picked up in the example of FIG. 31A. In this case, a point picked up by the image pickup means 1001 in a direction 1002 is a point (bumper of a vehicle) 2001 positioned above the three-dimensional road surface. However, a road surface is assumed in the case that an image is compounded from a virtual viewpoint 1004 so that image composition is made assuming that the point to be picked up is present on a point 2002 as an intersection of the direction 1002 and the road surface.
FIG. 32B shows an actual vehicle, a rear camera, and the point 2001 of a bumper position of a vehicle in the rear. FIG. 32C is an image of the image pickup apparatus 1001. In the image, the point 2001 of the bumper and a point 2002 of the intersection of its direction and the road surface are overlaid on a single point. In the composite image shown in FIG. 32D, the point 2002 is compounded farther than the point 2001. Moreover, other portions of the vehicle in the rear are compounded as if they existed farther than actual position and are distorted to a large extent, except the tires which come in contact with the road surface.
In this way, in the related art, only the ground is assumed for conversion of viewpoint. As a result, what is not on the three-dimensional ground, for example other vehicles and obstacles are distorted in a composite image.
When the related art is utilized as driving support apparatus, the bumper of another vehicle is displayed farther than the actual position. While the user driving a car assumes an ample distance from this image, the distance to the actual obstacle is longer so that the car is more likely to come in contact with the object. Thus, removing the dislocation and image distortion is an important problem in applications.
As a countermeasure against such distortion of a viewpoint-converted image from above, there is disclosed an example in JP-A-7-186833. In this example, a same color is extracted across the image, then a road surface area and a non-road-surface area are separated by the expansion and contraction of the area. For the road surface area, a converted image from above is compounded. For the non-road-surface area, an input image of the area is scaled up/down without performing viewpoint conversion and the resulting image is pasted on the converted image. This allows composition of an image without distortion of obstacles present above the road.
However, this example leaves the problems described in the problems (1) through (3) below.
(1) Separation of a road surface area and non-road-surface area is made using color information, so that separation is inaccurate at a portion where texture changes to a great extent on the road surface and a portion of a similar color to that of a road, such as a building.
(2) An obstacle apart from the ground, such as a bumper is compounded as part of a road. This extends the road area farther than the actual road in the composite image.
(3) While the road surface is converted to an image from a viewpoint above, the obstacle remains an input image. The resulting composite image is unnatural and the driver may have difficulty in intuitively understand the information on the surrounding.
Concerning the problems (1), a technology to separate a road surface and a non-road surface is disclosed in JP-A-7-334679. In this example, images picked up by right and left cameras are associated to coincide at a position where they are projected onto a road surface. Then separation is made: an area where the two corresponding image signals are similar to each other within a threshold is defined as a road surface area and the remaining area as a non-road-surface area.
This example, however, leaves the problems (1) through (3) below.
(1) An obstacle apart from the ground, such as a bumper, is recognized as part of a road farther than the actual road.
(2) Vertical edges are easy to recognize from stereo cameras arranged right and left although portions without edges and horizontal edge portions cannot be recognized. In particular, the boundary with an obstacle apart from the ground such as a bumper tends to appear as a horizontal edge on the screen.
(3) While viewpoint conversion from above is not mentioned in this example, there is no effect of distortion correction on a converted image of an obstacle.
The invention has been proposed in order to solve the problems of the related art driving support apparatus and aims at providing driving support apparatus which allows the driver to intuitively and more accurately recognize the situation around an obstacle thus reducing a burden of the driver.